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SECTION  1 


INTRODUCTION 


1.1  PURPOSE 

This  document  provides  the  test  plan  and  guidelines  for  the  experimental  eval¬ 
uation  of  video  transmissions  of  radiographs  over  the  Remote  Medical  Diagnosis  System 
(RMDS)  Advanced  Development  Model  (ADM)  terminals.  The  purpose  of  this  evaluation 
was  to  determine  which,  if  any,  of  the  various  operational  modes  of  the  RMDS  ADM 
terminal  would  satisfy  the  image  fidelity  requirements  for  clinical  diagnosis  of  video- 
transmitted  radiographs.  The  objective  of  this  evaluation  was  to  establish  quantitative 
and  qualitative  values  or  relationships  delineating  the  image  fidelity  requirements 
necessary  for  professional  radiologists  to  make  correct  and  confident  diagnosis  of 
video-transmitted  radiographs. 


1.2  BACKGROUND 

The  mission  of  the  RMDS  is  to  improve  medical  diagnosis  at  remote  sites. 
This  is  accomplished  by  transmitting  medical  data  and  diagnostic  information  between 
remote  ship  or  shore  sites  and  full-capability  medical  centers.  The  RMDS  will  enable 
the  medical  personnel  at  a  remote  site  to  contact  a  physician  at  a  diagnostic  center 
(ashore  or  shipboard)  and  transmit  a  visual  and  auditory  presentation  of  the  medical 
data  needed  for  diagnosis,  such  as  patient  history,  laboratory  tests,  ECG  tracings,  x-ray 
images,  images  of  a  patient  injury,  heart-lung  sounds,  and  verbal  descriptions.  By 
return  link,  the  physician  will  be  able  to  send  diagnosis  and  treatment  information.  The 
communication  requirements  are  satisfied  by  any  two-way,  voice-grade,  narrowband 
communication  channel  such  as  telephone  line,  hf  or  uhf  radio,  or  satellite  link. 

The  system  as  a  whole  consists  essentially  of  the  RMDS  terminals,  the  existing 
voice-grade  communication  links  used  to  interconnect  the  terminals,  and  user  person¬ 
nel.  Contained  in  the  terminals  is  all  the  hardware  that  is  unique  to  the  system:  TV 
camera,  TV  monitor,  x-ray  light  box,  electronic  stethoscope,  ECG  monitor,  audio  tape 
recorder,  audio  handsets,  and  the  electronics  package,  consisting  of  signal  modulator, 
demodulator,  and  modems. 

Shipboard  feasibility  tests  of  an  early  RMDS  were  completed  during  FY  75/76. 
This  testing  showed  that  the  concept  was  feasible  and  that  equipment  could  be  devel¬ 
oped  to  meet  the  requirements  by  using  available  technology  (ref  1).  Because  of  various 
constraints  (eg,  narrow  bandwidth,  short  transmission  times,  etc.),  the  resolution  and 
gray  scale  to  be  achieved  in  transmitting  and  displaying  radiographic  data  should  be 


1.  NOSCTR  659,  Feasibility  Tests  of  the  Remote  Medical  Diagnosis  System, 
WT  Rasmussen  and  I  Stevens  (NOSC)  and  JA  Kuhlman  (WESTEC  Services,  Inc.), 
January  1981. 
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kept  to  a  minimum  to  meet  essential  requirements.  The  gray  scale  requirements  for 
display  have  been  satisfactorily  established  in  existing  literature  to  be  at  least  6  bits 
per  picture  element,  with  8  bits  preferred.  In  order  to  derive  minimum  resolution  data, 
eight  radiologists  were  tested  and  the  results  analyzed  in  FY  77,  using  a  special  digital 
closed-circuit  television  system  to  simulate  the  RMDS  equipment  (ref  2). 

As  a  result  of  those  feasibility  and  radiology  tests,  NOSC  undertook  a  develop¬ 
ment  project  to  produce  two  Advanced  Development  Model  RMDS  terminals;  the  ADMs 
were  procured  in  September  1977.  The  RMDS  ADM  terminals  were  tested  for  technical 
performance  at  NOSC  from  September  1977  to  September  1979.  During  this  period,  the 
ADMs  were  tested  at  sea,  with  one  terminal  aboard  the  USS  ENTERPRISE  (CVN-65) 
from  28  February  to  5  March  1978.  The  image  fidelity  evaluation  testing  of  radiograph 
transmission  via  the  ADMs,  per  this  test  plan,  was  performed  during  the  period  October 
1978  to  April  1979.  The  at-sea,  laboratory,  and  radiology  test  results  are  documented 
in  separate  NOSC  Technical  Reports. 


1.3  EXPERIMENTAL  CONSTRAINTS  AND  CONSIDERATIONS 

a)  The  tests  had  to  be  structured  for  minimum  impact  on  the  radiologists' 
professional  duties,  without  detrimental  impact  on  good  experimental 
design  and  procedural  practices. 

b)  The  tests  had  to  provide  sufficient  data  for  statistically  valid  analyses 
to  support  the  conclusions  reached. 

c)  The  tests  had  to  conform  to  the  available  and  existing  equipments. 

d)  The  validated  test  results  obtained  from  this  effort  were  utilized  as 
some  of  the  principal  inputs  in  developing  an  RMDS  procurement  spec¬ 
ification. 


1.4  PARAMETRIC  CONSIDERATIONS 

The  successful  (ie,  accurate  and  confident)  diagnosis  of  a  radiographic  image 
by  a  professional  radiologist  depends  almost  exclusively  on  the  fidelity  of  the  image 
viewed.  The  fidelity  of  radiographs  transmitted  electronically  from  one  location  to 
another  is  influenced  considerably  by  various  parameters  such  as  transmission  mode, 
equipment  characteristics,  etc.  Under  test  were  the  parameters  that  affect  image 
fidelity,  as  follows: 

a)  Gray  level  quantization,  which  can  make  natural  gray  level  changes 
appear  as  artificial  edge  structures,  or  which  can  mask  subtle  gray 
level  changes  (ie,  analog  vs.  digital  transmission  mode). 


2.  NOSC  TR  150,  Resolution  Requirements  for  Slow-Scan  Television  Transmission  of 
X-rays,  FH  Gerber,  19  September  1977. 
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b) 


Additive  noise  in  the  transmission  signal,  which  gives  a  random,  tex¬ 
tured  pattern  to  the  image,  thereby  possibly  masking  natural  texture 
or  detail  (ie,  high  vs.  low  signal-to-noise  ratio). 
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c)  Spatial  resolution  of  the  image  presented,  which  affects  the  level  of 
detail  that  can  be  detected  under  low  noise  conditions  (ie,  fine  vs. 
coarse  resolution). 

The  tests  performed  were  directed  toward  resolving  both  the  nature  and  the 
impact  of  these  three  parameters  on  radiologists'  diagnostic  performance  in  evaluating 
transmitted  radiographs.  In  addition  to  the  three  principal  parameters,  the  fidelity  of 
the  transmitted  images  may  be  affected  by  the  settings  of  the  contrast  and  brightness 
controls  at  the  TV  monitor.  Therefore,  any  changes  in  these  settings  made  by  the 
radiologists  were  monitored  and  recorded  during  the  tests. 


1 
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SECTION  2 


APPROACH  AND  DESIGN 


2.1  EXPERIMENTAL  APPROACH  AND  PARAMETERS 

The  experimental  approach  taken  involved  the  use  of  two  groups  of  test  sub¬ 
jects  (radiologists):  an  RMDS  Radiology  Test  Group  and  a  Control  Radiology  Test 
Group.  The  test  data  from  each  group  were  analyzed  and  compared  individually  as  well 
as  by  groups.  In  the  following  two  subsections,  the  experimental  variables  considered 
for  each  group  are  listed. 

2.1.1  RMDS  Radiology  Tests  Experimental  Variables 

Each  of  the  three  variables  below  has  two  levels: 

a)  Transmission  Mode: 

•  Digital  (64  shades  of  gray)  Transmission 

•  Analog  Transmission 

b)  Signal-to-Noise  Ratio  (SNR): 

•  High  SNR 

•  Low  SNR 

c)  Resolution: 

•  Fine  (256  x  512  raster)  Resolution 

•  Coarse  (256  x  256  raster)  Resolution 

Combining  the  variables  and  levels  in  a  true  factorial  design*  would  result 
in  23  experimental  conditions.  However,  since  there  is  no  progressive  degradation  of 
SNR  in  digital  transmission,  the  theoretical  combinations  of  digital  transmission  and 
low  SNR  (digital  x  low  SNR  x  fine  resolution;  digital  x  low  SNR  x  coarse  resolution)  are 
not  valid.  Therefore,  the  RMDS  Radiology  Tests  consisted  of  the  six  experimental 
conditions  shown  in  table  1. 


*In  a  factorial  design  the  effects  and  interactions  of  two  or  more  experimental  variables 
are  observed  simultaneously. 
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TRANSMISSION  MODE 

ABBREVIATION 

LABEL 

Digital;  High  SNR;  Fine  Resolution 

DHF 

i  ; 

Digital;  High  SNR;  Coarse  Resolution 

DHC 

ii 

Analog;  High  SNR;  Fine  Resolution 

AHF 

iii 

Analog;  High  SNR;  Coarse  Resolution 

AHC 

IV 

Analog;  Low  SNR;  Fine  Resolution 

ALF 

V 

Analog;  Low  SNR;  Coarse  Resolution 

ALC 

VI 

Table  1.  Six  RMDS  radiology  experimental  test  conditions. 


2.1.2  Control  Radiology  Tests  Experimental  Variables 

These  tests  were  considered  a  necessary  adjunct  to  the  RMDS  Radiology 
Tests,  and  compared  the  differences  in  findings  confidence  levels,  diagnostic  confidence 
levels,  and  accuracy  of  radiographs  presented  in  the  following  two  manners:  as  a  "pure" 
analog  signal  (not  possible  with  the  RMDS*)  via  closed  circuit  television  (CCTV);  and  in 
the  "direct"  manner,  with  a  light  box.  Resolution  of  the  CCTV  image  was  made 
equivalent  to  that  of  the  fine  resolution  mode  of  the  RMDS  Tests  by  band-limiting  the 
video  signal.  The  SNR  was  also  fixed  for  the  above  two  variables.  Thus,  the  Control 
Tests  were  performed  under  two  experimental  conditions: 

•  Direct  -  I 

•  CCTV  -  II 


2.2  HYPOTHESES 

Two  principal  hypotheses  were  tested  by  statistical  analyses  of  the  data: 

a)  With  the  combinations  of  fine  resolution  and  high  SNR,  for  both  analog 
and  digital  transmission  modes,  diagnoses  can  be  made  which  are,  sta¬ 
tistically,  not  significantly  different  in  terms  of  confidence  level  and 
accuracy  from  those  made  using  CCTV  images  of  equivalent  spatial 
resolution.  In  other  words,  the  quantization  level  of  6  bits  per  picture 
element,  as  used  in  the  RMDS  for  digital  transmission,  and  the  frozen 
noise  of  the  received  images  for  both  digital  and  analog  transmission 
modes  degrade  neither  the  confidence  level  nor  the  accuracy  of 
radiographic  diagnoses. 


*Since  the  RMDS  utilizes  a  memory,  those  radiographs  transmitted  in  the  analog  mode 
are  digitized  to  a  certain  degree. 
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b)  Fine  resolution,  in  both  the  digital  and  analog  RMDS  rr,  »;• 

statistically  more  confident  and  accurate  diagnoses  than  ■■■■■ 
lution  and  significantly  reduces  diagnostic  inaccuracies. 

In  addition,  other  quantitative  results,  as  outlined  in  paragraph  3.3  below. 
gated  during  the  tests. 


2.3  TEST  REQUIREMENTS 

2.3.1  RMDS  Radiology  Test  Requirements 

a)  Radiographs 

Thirty-six  radiographs  were  used,  divided  into  six  set 
radiographs  each.  These  36  radiographs  were  selected  from  over  sixty  case  Pie- 
Naval  Regional  Medical  Center,  San  Diego.  The  six  radiographs  of  each  set  cor.s 
six  different  disorders,  with  all  sets  balanced  as  equivalently  as  possible  with 
pathology,  contrast,  density,  and  the  difficulty  of  visual  and  diagnostic  interpret. 

This  was  done  by  assigning  a  Difficulty  Ranking  Factor  (DRF)  of  1  (low)  to  6  b.a" 
each  radiograph  prior  to  its  inclusion  in  the  testing.  The  DRF  was  derived 
individuals,  independently,  by  consensus. 

The  six  sets  (labeled  A,  B,  C,  I),  E,  and  F)  were  then  arrange  - 
that  each  set  had  a  radiograph  with  a  DRF  of  1  through  6,  and  each  was  balam-e 
much  as  possible  with  respect  to  pathologies  and  type  (or  zone),  ie,  appem:*,,- 
abdomen,  chest,  or  skull.  The  sequential  order  of  presentation  of  the  radiograph  *  .v 
each  set  was  randomized  with  respect  to  the  DRF  and  type.  This  order  was  the 
for  all  subjects  throughout  the  testing.  Table  2  shows  the  radiographs  used,  their 
of  presentation  within  the  set,  the  type  of  radiograph,  the  assigned  DRF,  and  a  prr>  ar 
diagnosis. 


b)  Subjects 

Six  experienced  radiologists  were  required.  It  was  estimated  that  n 
to  two  hours  would  be  needed  by  each  radiologist  to  evaluate  and  diagnose  one  set  of 
radiographs,  including  time  required  to  record  findings  and  comments.  Based  on  t’n- 
assumption,  at  most  twelve  hours  would  be  required  by  each  radiologist  to  complete  tn. 
experiment. 

-  2.3.2  Control  Radiology  Test  Requirements 

a)  Radiographs 

All  six  radiograph  sets  (A,  B,  C,  D,  E,  and  F)  used  in  the  RMDS 
Radiology  Tests  were  employed.  The  six  radiographs  in  each  set  were  sequentially 
ordered  as  shown  in  table  2.  All  six  sets  were  presented  in  a  direct  manner  with  a  light 
box;  three  of  the  sets  were  first  presented  via  closed  circuit  television  (CCTV),  making 
a  total  of  nine  sets  of  evaluations. 
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Set/# 

DRF* 

Type  (Zone) 

Diagnosis 

A-l 

5 

Appendage 

Soft  tissue  hemangioma 

2 

6 

Skull 

Fractured  mandible 

3 

3 

Chest 

(R)  LL  pneumonia 

4 

2 

Skull 

Double  floor  of  sella 

5 

4 

Abdomen 

Prostatic  calculi 

6 

1 

Abdomen 

Bilateral  adrenal 
calcification 

B-l 

1 

Skull 

Broken  nose 

2 

3 

Appendage 

Osteoid  osteoma 

3 

6 

Chest 

ASD  with  4:1  shunt 

4 

2 

Chest 

Alveolar  cell  calcification 

5 

4 

Skull 

Intracranial  air  &  fracture 

6 

5 

Appendage 

Fx  neck  of  femur  on  (R) 

C-l 

4 

Chest 

Calcified  mitral  annulus 

2 

2 

Chest 

Cocci 

3 

5 

Skull 

Parietal  skull  fracture 

4 

1 

Abdomen 

Abdominal  aortic  aneurysm 

5 

3 

Appendage 

Avascular  necrosis  of  lunate 

6 

6 

Skull 

Nasal  spine  Fx 

D-l 

4 

Chest 

Pancoast  tumor 

2 

5 

Skull 

Multiple  myeloma 

3 

2 

Abdomen 

Air  under  (R)  diaphragm 

4 

6 

Appendage 

Chond  rocalc  inosis 

5 

1 

Appendage 

Fibrous  cortical  defect 

6 

3 

Chest 

Pericardial  calcification 

E-l 

3 

Skull 

Calvarial  hemangioma 

2 

6 

Abdomen 

Splenomegaly 

3 

4 

Appendage 

Tibial  stress  Fx 

4 

1 

Abdomen 

Osteitis  condensans  ilii 

5 

2 

Skull 

Mucocoele  (L)  fronal  sinus 

6 

5 

Chest 

Histioeytosis-x 

F-l 

3 

Abdomen 

Abdom.  calcification  (post 
traumatic  splenic  cyst) 

2 

2 

Chest 

Infectious  spondylitis  TB 

3 

4 

Appendage 

Cocci  osteomyelitis 

4 

5 

Chest 

Calcified  myocardial  infarct 

5 

6 

Chest 

Pneumothorax  on  (R) 

6 

1 

Skull 

Enlarged  sella 

*DRF  =  Difficulty  Ranking  Factor:  1  (Low),  2  (Low/Med  Low),  3  (Med  Low),  4  (Med 
High),  5  (Med  High/High),  6  (High) 


Table  2.  Six  sets  of  test  radiographs. 
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b)  Subjects 

Six  radiologists,  different  from  those  participating  in  the  RMDS 
Radiology  Tests  but  of  comparable  experience,  were  required.  It  was  estimated  that  up 
to  one-half  hour  would  be  needed  by  each  radiologist  to  diagnose  one  set  of  six 
radiographs,  so  that  four  and  one-half  hours  would  be  required  by  each  radiologist  to 
evaluate  the  nine  sets. 


2.4  EXPERIMENTAL  DESIGN 

2.4.1  RMDS  Radiology  Tests 

The  six  experimental  conditions  (paragraph  2.1.1,  table  1),  radiographs,  and 
subjects  were  combined  into  a  modified  factorial  treatments-by-subjects*  design,  opti¬ 
mized  for  isolating  the  effects  of  the  variables  to  be  tested.  Table  3  shows  this  design. 
Its  pertinent  features  are: 

a)  It  was  balanced  for  subjects,  conditions  and  radiograph  sets,  so  that 
each  subject  was  tested  once  under  each  condition  with  a  different 
set  of  radiographs. 

b)  No  set  of  radiographs  was  paired  with  any  one  experimental  condi¬ 
tion  more  than  once. 

c)  The  sequence  of  conditions/sets  presentations  was  different  for  each 
of  the  six  subjects,  with  respect  to  both  experimental  conditions  and 
radiograph  sets. 

This  type  of  design  minimized  any  possible  undesirable  interactions  between  radio¬ 
graphs,  subjects,  and  sequence  of  experimental  conditions  presentation. 

2.4.2  Control  Radiology  Tests 

In  the  Control  Tests,  all  six  sets  of  radiographs  were  used.  Each  subject  was 
tested  under  experimental  conditions  I  and  n  with  the  appropriate  radiograph  sets  and  in 
the  order  shown  in  table  4. 


♦In  a  treatments-by-subjects  design,  all  experimental  conditions  are  successively  admin¬ 
istered  to  the  same  subjects. 


1 


TESTING  sequence 


Table  4.  Control  Test  check  form 


SECTION  3 


METHOD  AND  PROCEDURE 


3.1  GENERAL 

Both  RMDS  Radiology  and  Control  Tests  were  carried  out  in  the  Bioengi¬ 
neering  facilities,  Code  5123,  Naval  Ocean  Systems  Center  (NOSC).  Two  RMDS  termi¬ 
nals  located  in  the  area  were  used  back-to-back,  with  one  system  serving  as  the  remote 
terminal  (transmitter)  and  the  other  one  as  the  diagnostic  center  terminal  (receiver). 
Voice  communication  is  one  feature  of  the  RMDS,  and  it  was  available  and  used  in 
testing.  The  remote  terminal  provided  zoom  and  selective  intensity  capabilities.  In  the 
Control  Tests,  the  RMDS  was  bypassed  and  the  radiographic  image  as  seen  by  the  TV 
camera  was  displayed  directly  on  the  subject’s  TV  monitor  via  a  bandpass  filter. 
Figure  1  is  a  block  diagram  of  the  RMDS  equipment  experimental  configuration.  A 
cassette-type  tape  deck  was  used  for  recording  any  comments  made  by  the  subjects 
throughout  the  duration  of  each  test  session. 

In  order  to  insure  that  each  radiograph  was  repeatedly  displayed  on  the  receive 
TV  monitor  to  each  radiologist  at  the  same  brightness  and  contrast,  some  of  the  levels, 
namely  the  camera  lens  aperture  opening  (K),  the  video  level  (VL),  and  the  black  level 
(BL)  on  the  ship  transmit  RMDS  terminal,  were  recorded  in  advance  in  terms  of  the 
amount  of  deflection  measured  on  a  strip-chart  recorder.  The  settings  were  determined 
in  such  a  way  that  the  image  displayed  on  the  receive  TV  monitor  was  as  close  as 
possible  to  the  actual  radiograph  image;  table  5  shows  these  settings.  Thus,  throughout 
the  tests  the  settings  in  table  5  were  used  and  recorded.  Furthermore,  the  contrast  and 
brightness  levels  of  the  receive  TV  monitor  were  also  monitored  and  recorded.  The 
monitoring  of  the  last  two  variables  gave  an  indication  as  to  what  the  radiologist  did  to 
the  contrast  and  brightness  on  the  receive  monitor  in  order  to  make  a  diagnosis.  Prior 
to  each  testing  session,  all  equipments  were  thoroughly  checked  for  satisfactory  perfor¬ 
mance  of  all  necessary  functions,  and  adjusted,  if  required,  to  baseline  operating  para¬ 
meters. 


Prior  to  testing,  each  of  the  radiologists  participating  in  the  experiment  was 
briefed*  on  the  purpose  of  the  RMDS;  the  reasons  for  and  objectives  of  the  test;  the 
capabilities,  features  and  limitations  of  the  hardware  relevant  to  the  experiment  (such 
as  the  zoom  and  selective  intensity  capabilities,  etc.);  the  test  facility  and  procedures; 
and  the  test  materials  such  as  the  test  data  sheets  used  for  both  Test  and  Control 
Groups  (shown  as  figure  2).  The  participants  were  told  that  their  diagnostic  comments 
would  be  tape  recorded,  and  they  were  given  an  opportunity  to  familiarize  themselves 
with  the  equipment,  the  facilities,  and  the  task  to  be  performed,  using  three  non-test 
radiographs.  The  radiologists  did  not  know  to  which  group  they  belonged. 


j  *See  Appendix  A  for  full  text  of  radiologists'  briefing. 

I 

I 
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Figure  1.  RMDS  radiology  test  equipment  configuration 


cn  4*  co  to  >-»  oj  tn  w  to  m  o>  cn  ^  w  to  m  o)  cn  w  to  h  oj  cn  ^  w  to  h  O)  cn  ^  co  to  h 


MFE 

K  (mm) 

(Camera  Lens 

GOULD-4 

GOULD-3 

Aperture 

VL  (mm) 

BL  (mm) 

Set/# 

Setting) 

(Video  Level) 

(Black  Level) 

27 

24.5 

22.5 

16 

20 

18 

28 

24 

20 

26 

23.5 

12 

28 

23.5 

15 

28 

23.5 

15 

27 

25 

16 

27 

26.5 

16 

28 

26.5 

16 

28 

24 

20 

20 

24 

12 

28 

23.5 

16 

28 

21 

25 

28 

24 

19 

27 

23 

14 

28 

21.5 

17.5 

18 

20 

17 

27.5 

21 

20 

27 

20 

26 

27 

21 

18 

28 

21 

22 

27 

23 

14 

27 

23 

14 

28 

23 

14 

27 

18 

20 

28 

19 

22 

27 

19.5 

22 

27.5 

17 

22 

26 

15 

25 

28 

20 

25 

28 

20 

26.5 

27 

11.5 

28 

27.5 

16 

27 

27.5 

13 

27 

28 

25 

23 

27 

20 

21 

Table  5.  Strip  chart  recorder  settings. 
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In  order  to  minimize  any  possible  bias  on  the  part  of  the  subjects,  the  specific 
experimental  condition  under  test  at  any  one  time  was  not  revealed.  The  fact  that 
different  video  transmission  modes  for  radiographic  transmissions  were  being  investi¬ 
gated  was  included  in  the  introductory  briefing.  During  this  briefing,  it  was  emphasized 
that  the  objectives  of  the  experiment  were  not  to  evaluate  individual  professional  per¬ 
formance  and  capabilities,  but  only  to  determine  and  assess  the  physical  parameters 
necessary  for  the  effective  reading  and  evaluation  of  transmitted  radiographs.  The 
participating  radiologists  were  instructed  to  read  each  radiograph  as  rapidly  as  possible 
consistent  with  professional  responsibilities,  and  to  exclude  time  as  a  parameter  under 
investigation.  The  subjects  were  instructed  to  record  on  their  test  data  sheets  a 
numerical  value  for  their  level  of  confidence  in  the  findings  and  diagnoses  arrived  at  for 
each  radiograph  viewed.  These  values  could  range  from  1  for  a  low  confidence  level  to 
5  for  a  high  confidence  level.  The  subjects  were  also  encouraged  to  record  their  com¬ 
ments  in  the  space  provided  on  the  data  sheet.  Depending  on  whether  the  subject  was 
part  of  the  Test  or  Control  Group,  the  corresponding  RMDS  test  sequence  was  fol¬ 
lowed.*  The  radiologists  were  given  a  break  after  completion  of  each  set. 

No  radiologist  was  allowed  to  consult  with  others  during  the  course  of  testing. 
The  radiologists  were  also  requested  not  to  discuss  observations,  impressions,  or  diag¬ 
noses  with  the  other  radiologists  participating  in  the  study  until  completion  of  the 
entire  experiment.  After  completion  of  all  tests  the  radiologists  filled  out  a  question¬ 
naire,**  in  which  each  could  express  likes,  dislikes,  and  suggested  areas  of 
improvement. 

Radiologists'  findings  and  diagnoses  obtained  by  direct,  light  box  viewing  of 
each  radiograph  were  reviewed  by  Dr.  FH  Gerber,  Head  of  Department  of  Radiology, 
Naval  Regional  Medical  Center,  San  Diego,  California.  Findings  and  diagnoses  which 
were  agreed  upon  by  a  consensus  of  the  Control  Group  were  used  as  a  standard  against 
which  the  findings  and  diagnoses  of  the  RMDS  Radiology  Test  Group  were  compared. 
Thus,  the  standard  for  each  radiograph  was  entered  on  the  proper  forms  (figures  3 
and  4).  Dr.  Gerber  compared  each  radiologist's  findings  and  diagnoses  to  the  standard, 
and  established  an  Overall  Clinical  Reading  (OCR)  evaluation  of  acceptable,  marginal, 
or  unacceptable  for  each  radiograph  in  each  mode. 


3.2  DATA  ANALYSIS 
3.2.1  Quantitative  Analysis 

The  data  base  for  the  mathematical  treatment  of  the  test  results  consisted 
of  the  numerical  confidence  level  scores,  the  Overall  Clinical  Reading  evaluations,  and 
the  Difficulty  Ranking  Factors.  The  first  are  subjective  scores,  arrived  at  by  the 
subjects  themselves,  while  the  Overall  Clinical  Reading  scores  are  an  objective  measure 
of  performance.  The  Difficulty  Ranking  Factors  assess  the  difficulty  of  visual  and 
diagnostic  interpretation. 


*See  Appendix  B  for  details  of  the  RMDS  test  sequences. 

**See  Appendix  C  for  copy  of  questionnaires. 


Ill 


Figure  3.  Radiology  data  collection  sheet  for  RMDS  Test  Group, 


Figure  4.  Radiology  data  collection  sheet  for  Control  Group 


The  experimental  design  (modified  factorial  and  conditions  by  subjects)  as 
well  as  the  performance  criteria  (confidence  levels,  percentage  errors,  and  difficulty 
ranking)  allowed  application  of  mathematical  analysis.  The  statistical  analyses  per¬ 
formed  on  the  interval-scale,  confidence  level  data  included:  one-way  analysis  of  vari¬ 
ance  (ANOVA),  Duncan's  multiple  range  test,  Pearson's  correlation,  and  Wilcoxon's 
matched-pairs  signed-ranks  test.  The  tests  used  to  analyze  the  ordinal-scale,  OCR  data 
included:  Wilcoxon's  matched-pairs  signed-ranks  test  and  Chi-square.  The  DRF  mea¬ 
surements  fall  between  the  ordinal  and  interval  scales  of  data;  DRF  data  were  utilized 
for  correlation  purposes,  via  Pearson's  correlation,  and  as  subheadings  for  data 
groupings.  By  applying  the  proper  statistical  treatments  to  the  raw  data,  the  following 
issues  were  investigated: 

a)  RMDS  Radiology  Test  Group 

1)  Evaluate  Overall  Clinical  Reading  results  by  mode. 

•  Significant  differences. 

•  Intrasample  variance. 

2)  Evaluate  Overall  Clinical  Reading  results  by  radiologist. 

•  Significant  differences. 

•  Intrasample  variance. 

3)  Test  differences  between  transmission  modes  with  respect 
to  confidence  levels. 

4)  Determine  correlation  between  confidence  levels  and  Over¬ 
all  Clinical  Reading  for  each  transmission  mode. 

5)  Evaluate  the  relationships  between  the  types  of  pathology, 
confidence  levels,  and  transmission  modes. 

6)  Evaluate  the  effects  of  the  use  of  the  zoom  on  the  Overall 
Clinical  Readings. 

b)  Control  Radiology  Test  Group 

1)  Test  difference  between  results  of  light  box  and  TV  monitor 
viewing. 

2)  Based  on  the  Difficulty  Ranking  Factor  (DRF)  of  the  radio¬ 
graphs,  repeat  the  above  test. 

3)  Test  the  intrasample  variance  between  radiologists. 
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3.2.2 


Qualitative  Analysis 


The  qualitative  data,  represented  by  photographic  representations  of  the 
received  radiographic  images,  by  graphs  of  the  collected  data,  by  the  brief  written 
comments  on  the  individual  test  data  sheets,  and  by  the  questionnaire  filled  out  by  each 
radiologist,  were  used  as  an  important  corollary  to  the  quantitative  data  in  arriving  at 
selected  performance  criteria  for  future  Remote  Medical  Diagnosis  Systems.  The 
importance  of  a  qualitative  assessment  of  the  documented  radiographic  evaluation  pro¬ 
cess  cannot  be  underestimated,  since  the  diagnosing  radiologist  at  the  diagnostic  center 
terminal  will  be  the  end  user  of  the  RMDS.  Therefore,  the  radiologists'  reactions  to 
RMDS  functional  capabilities  and  performance  were  of  prime  consequence.  Analysis  of 
the  qualitative  data  addressed  areas  such  as  the  following: 

a)  Determine  quality  of  received  images  by  visual  examination  of  pho¬ 
tographs,  by  transmission  mode. 

•  Full  scale  images 

•  Zoom  images 

b)  Summarize  confidence  and  Overall  Clinical  Reading  scores  using 
mean  and  standard  deviation,  by  transmission  mode. 

c)  Evaluate  use  of  zoom  controls  by  mode,  testing  sequence,  radiolo¬ 
gist,  and  Overall  Clinical  Reading. 

d)  Establish  mean,  maximum,  and  minimum  times  required  to  diagnose 
one  set  of  six  radiographs,  by  transmission  mode. 

e)  Characterize  test  radiographs  by  DRF,  zone,  and  pathology. 

f)  Analyze  radiologists'  comments  on  test  data  sheets. 

g)  Analyze  radiologists'  comments  on  questionnaire. 


3.3  FINDINGS 

At  the  conclusion  of  the  data  analyses,  a  comprehensive  test  report  was  pre¬ 
pared  which  organized  and  summarized  the  quantitative  and  qualitative  analyses  and 
results  in  the  form  of  narrative,  tables,  and  figures  (ref  3).  That  report  also  includes  a 
section  on  recommended  functional  and  performance  parameters  for  digitization, 
resolution,  and  SNR,  to  be  input  to  an  RMDS  procurement  specification.  The 


3.  NOSC  TR  683,  Remote  Medical  Diagnosis  System  (RMDS)  Advanced  Development 
Model  (ADM)  Radiology  Performance  Test  Results,  WT  Rasmussen,  PD  Hayes, 
I  Stevens  (NOSC),  FH  Gerber  (NRMC  San  Diego),  JA  Kuhlman  and  FW  Hutzelman 
(WESTEC  Services,  Inc.),  in  process  for  publication. 
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quantitative  and  qualitative  findings  and  interpretations  within  that  report  address  the 
following: 


a)  Indicate  acceptance  or  rejection  of  the  hypotheses  (section  2.2). 

b)  Provide  quantitative  measures  of  the  effects  of  each  experimental 
variable  on  diagnostic  performance,  independent  of  the  other  vari¬ 
ables.  For  example,  does  the  SNR,  by  itself,  have  a  statistically 
significant  influence  on  diagnostic  performance?  Which  variable(s),  if 
any,  has  (have)  the  most  profound  effect  on  diagnostic  performance? 
Quantize  the  level  of  influence  exerted  by  the  variable(s). 

c)  Determine  statistically  significant  differences  in  diagnostic  perfor¬ 
mance  between  each  of  the  various  transmission  modes  and  the  stand¬ 
ard  method. 

d)  Point  out  any  pathologies  (of  those  provided)  which  do  not  lend  them¬ 
selves  to  confident  and  accurate  diagnoses  from  transmitted  radio¬ 
graphs. 

e)  Provide  guidelines  to  the  performance  parameters  of  digitization, 
resolution,  and  SNR  for  an  RMDS  procurement  specification. 
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SECTION  4 


RESOURCES 


4.1  MATERIAL  RESOURCES 

The  following  resources  were  required: 

a)  Six  sets  of  six  radiographs  each  (a  total  of  36),  including  templates  for 
setting  camera  field  of  view  for  each  radiograph. 

b)  Two  ADM  RMDS  terminals  (one  each,  transmit  and  receive).  The  two 
RMDS  terminals  were  wired  back-to-back  so  that  they  could  readily  be 
switched  between  analog  and  digital  transmission  modes.  The  link 
included  the  circuitry  for  adding  noise  to  the  analog  transmission  line. 
In  addition,  the  transmitting  RMDS  unit  was  equipped  with  monitoring 
instrumentation  for  specific  and  precise  setting  of  the  TV  camera 
aperture  (brightness),  transmit  black  level,  and  video  level. 

c)  TV  monitor  (17-inch)  on  desk  or  table. 

d)  Voice  link  between  monitor  location  and  the  transmit  RMDS  room. 

e)  Cassette  tape  recorder  and  tapes  for  recording  test  proceedings. 

f)  Four-channel  strip  chart  recorder  to  monitor  subjects'  t  'ightness  and 
contrast  adjustments  on  TV  monitor,  and  to  record  transmit  black 
level  and  video  level  settings. 

g)  Direct  link  from  RMDS  TV  camera  to  TV  monitor  (for  CCTV  mode)  via 
appropriate  bandpass  filter. 

h)  Radiograph  light  box. 

i)  Subjects'  Test  Data  Sheets,  with  typewritten  case  history  included. 

j)  Observers'  data  collection  sheets  (for  both  RMDS  and  Control  Tests). 

k)  Experimental  scenario,  instructions,  and  instrumentation  operation 
instructions. 


4.2  PERSONNEL  RESOURCES 

Each  test  session  involved  one  radiologist  and  two  RMDS  transmission  equip¬ 
ment  operators.  Additionally,  a  technician  familiar  with  the  RMDS  equipment  was 
available  to  resolve  any  equipment  problems. 
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4.3 


SCHEDULE 


4.3.1  RMDS  Radiology  Tests 

The  radiologists  participating  in  the  tests  were  scheduled  at  random,  one  per 
each  3-  to  4-hour  afternoon  session,  according  to  the  radiologists'  availability.  The 
length  of  each  session  depended  on  how  fast  the  radiologist  could  diagnose  the  radio¬ 
graphs.  Each  session  included  two  or  three  trials.  A  test  session  did  not  end  unless  a 
trial  had  been  completed.  Each  trial  corresponded  to  a  particular  transmission  mode 
and  included  six  radiographs  presented  to  the  radiologist.  Thus,  each  radiologist 
completed  the  test  in  two  to  three  sessions. 

4.3.2  Control  Radiology  Tests 

Radiologists  were  scheduled  in  the  same  way  as  in  4.3.1,  and  test  sessions 
were  about  as  long.  There  was  a  total  of  nine  trials  per  radiologist,  and  each  session 
consisted  of  four  to  five  trials.  Thus,  each  radiologist  completed  the  test  in  two 
sessions. 


APPENDIX  A 

RADIOLOGISTS'  INTRODUCTION 
TO  RADIOLOGY  TESTING 


Prior  to  the  first  test  session,  each  radiologist 
was  briefed  on  the  purpose  of  the  RMDS;  the 
reasons  for  and  the  objectives  of  the  test;  the 
capabilities,  features,  and  limitations  of  the 
hardware;  the  test  materials;  and  the  test  pro¬ 
cedures.  The  following  introductory  material 
was  given  to  and  discussed  with  each  radi¬ 
ologist. 
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INTRODUCTION  TO  RADIOLOGY  TESTING 


This  is  a  Remote  Medical  Diagnosis  System  (RMDS),  whose  general  mission  is  to 
improve  medical  diagnosis  at  remote  sites  (ship  or  shore)  by  exchanging  medical  data 
and  diagnostic  information  between  these  remote  sites  and  full-capability  medical  cen¬ 
ters.  The  RMDS  will  enable  the  medical  personnel  at  a  remote  site  to  contact  a 
physician  at  a  diagnostic  center,  either  ashore  or  shipboard,  and  transmit  the  medical 
data  needed  for  diagnosis  sue*  as  patient  history,  laboratory  tests,  ECG  tracings,  x-ray 
images,  images  of  a  patient  injury,  heart-lung  sounds,  and  verbal  descriptions.  By 
return  link,  the  physician  will  be  able  to  send  diagnosis  and  treatment  information.  The 
communication  requirements  for  this  are  satisfied  by  any  two-way  voice-grade  nar¬ 
rowband  communication  channel  such  as  telephone  line,  hf  or  uhf  radio,  or  satellite 
links. 


You  are  going  to  participate  in  a  test  to  help  determine  video  requirements  for 
the  transmission  of  x-ray  images.  TTiis  unit  will  be  used  as  the  receive  unit,  and  the 
transmit  unit  is  in  the  room  next  door.  However,  both  units  are  identical  and  can  be 
used  for  either  transmitting  or  receiving. 

We  are  now  going  to  give  you  three  example  cases  to  demonstrate  and  familiarize 
you  with  the  system.  The  first  case  I  will  run  through  quickly,  but  the  second  and  third 
cases  will  be  handled  entirely  by  you  and  I  will  assist  or  answer  any  questions  you  may 
have. 

1.  You  will  be  provided  a  documentation  sheet,  such  as  these,  for  each  case 
in  your  test  session.  Some  general  history,  symptomatic,  or  clinical  data 
may  be  given.  You  are  asked  to  record  your  findings  and  diagnoses,  and 
indicate  a  level  of  confidence:  l=low  to  5=high. 

2.  You  will  see  the  image  of  the  x-ray  on  the  monitor  on  the  table.  (For 
Control  Group:  You  will  also  be  asked  to  view  x-ray  films  using  the  light 
box  on  the  wall.)* 

3.  You  may  adjust  the  contrast  and  brightness  of  the  monitor  to  suit  your¬ 
self.  Tliese  settings  are  being  recorded.  In  addition,  members  of  the  Test 
Group  will  have  the  option  of  viewing  the  images  in  either  positive  or 
negative  polarity. 

4.  If  you  want  a  close-up  of  any  portion  of  the  image,  just  tell  the  operator 
in  the  other  room  via  the  audio  link  you  have.  Use  the  plastic  overlay  on 
the  monitor,  and  advise  the  operator  which  of  the  nine  areas  or  portions 
thereof  you  would  like  enlarged.  (For  Test  Group:  If  you  request  a 
close-up  image,  you  may  retain  the  full-size  image  on  this  second  monitor 
by  pushing  this  FRAME  FREEZE  button.)* 


♦These  statements  were  included  in  the  introduction  to  the  Control  Group  and  Test 
Group,  respectively,  but  subjects  were  not  advised  as  to  which  group  they  belonged  to. 
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5. 


The  ''idio  link  between  the  two  rooms  will  hIJoa  vou  to  nave  iwo-wa., 
voie"  "onversation  with  the  operator  if  you  have  any  prooi-111*  or  ques¬ 
tions.  However,  the  operator  will  not  provide  you  witn  any  additional 
data  on  the  test  cases.  Also,  you  will  be  tape-recorded  for  documentation 
purposes  during  the  test  sessions. 

6.  Please  don't  push  or  adjust  any  other  knobs  on  the  monitor  or  the  RMDS 
unit.  Everything  has  been  previously  set  for  each  test  session. 

7.  After  the  three  example  cases,  test  cases  will  be  shown  to  you  in  a  set  of 
six.  These  will  be  shown  to  you  in  a  fixed  order,  and  you  will  have  a 
corresponding  set  of  documentation  sheets. 

8.  You  will  not  be  led  astray  on  these  cases.  This  is  a  test  of  the  system,  not 
a  test  of  your  abilities.  If  you  have  any  additional  comments,  please  write 
them  down. 

AFTER  THE  EXAMPLE  CASES: 

9.  The  next  session  of  six  cases  may  take  approximately  one-half  hour  to  two 
hours  to  complete.  This  is  due  to  the  time  required  for  the  operator  to 
set  up  and  document  the  transmission  at  specific  settings  for  consistency 
among  all  testing  sessions.  After  the  first  session  there  will  be  a  break 
and  then  a  second  session. 

10.  Please  do  not  discuss  your  testing  session,  x-ray  cases,  or  opinions  of  the 
system  with  any  of  your  colleagues  until  after  all  testing  has  been  com¬ 
pleted. 


APPENDIX  B 

TEST  AND  CONTROL  SEQUENCE 
FOR 

RMDS  RADIOGRAPHIC  ANALYSIS 


The  following  is  a  sequence  of  steps  used  during 
the  testing  to  ensure  that  the  image  of  an  x-ray 
was  repeated  as  identically  as  possible  for  each 
test  subject. 
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TEST  AND  CONTROL  SEQUENCE 


Prior  to  any  testing,  radiographs  were  individually  reviewed.  The  RMDS  terminal 
adjustments  were  recorded  for  each  radiograph.  This  allowed  each  radiograph  to  be 
presented  identically  to  each  subject.  When  the  RMDS  adjustments  were  recorded  for 
each  radiograph,  a  major  consideration  of  the  NOSC  personnel  was  that  the  image 
presented  be  as  near  to  the  original  radiograph  as  possible;  ie,  there  was  no  attempt 
made  to  enhance  any  portion  of  the  images.  The  following  is  the  basic  sequence  used 
for  both  Test  and  Control  Groups: 

1.  Before  the  radiologist  arrived  for  testing,  the  RMDS  was  turned  on  and 
tested  to  ensure  proper  working  order. 

2.  On  the  radiologist's  first  test  period,  a  brief  background  was  presented  to 
familiarize  the  radiologist  with  the  RMDS.  During  all  test  periods,  the 
tape  recorder  in  the  Doctor's  room  was  turned  on  and  the  door  to  the 
room  was  closed  to  prevent  interference  and  distraction.  Once  the  radiol¬ 
ogist  was  isolated,  the  only  communication  was  through  a  push-to-talk 
hand  set. 

3.  In  an  adjacent  room,  an  RMDS  operator  set  up  a  radiograph  according  to 
the  previously  determined  adjustments  of  brightness,  contrast,  and  x-ray 
size.  When  the  image  was  correct  in  reference  to  the  prerecorded  adjust¬ 
ments,  a  picture  of  the  image  was  taken  for  reference  by  a  monitor  cam¬ 
era. 

4.  For  Test  Group  subjects,  the  image  was  transmitted  (in  the  appropriate 
mode)  to  the  receive  RMDS  terminal.  However,  the  image  was  withheld 
from  display  on  the  receive  monitor  during  the  transmission  time  to 
prevent  the  radiologist  from  seeing  the  slow-scan  image  development. 
When  the  radiologists  indicated  they  were  ready  for  another  case,  the  full 
image  was  switched  to  their  monitor.  The  only  adjustments  available  to 
the  radiologists  were  the  contrast  and  brightness  controls  on  their  mon¬ 
itor. 

5.  When  the  radiologist  signaled  completion  of  diagnosis,  the  image  on  the 
remote  monitor  was  switched  off  and  recordings  were  made  of  any  radiol¬ 
ogist  adjustments  to  the  contrast  and  brightness  of  the  remote  monitor. 

There  were  two  variations  to  the  basic  sequence  above:  one  was  the  zoom 
sequence  and  the  second  was  the  Control  I  sequence.  The  zoom  sequence  repeated 
steps  3  through  5  with  the  exception  that  in  step  3  the  adjustments  of  size,  brightness 
and  contrast  were  not  prerecorded.  This  allowed  for  some  enhancement  when  requested 
by  the  radiologist.  The  Control  I  sequence  used  the  light  box,  during  which  the  radiol¬ 
ogists  were  given  the  radiographs  one  at  a  time  until  the  diagnoses  were  completed. 


APPENDIX  C 


RADIOLOGISTS'  QUESTIONNAIRE 
FOLLOWING  RADIOLOGY  TESTING 


After  the  completion  of  all  tests,  each  radiol¬ 
ogist  was  asked  to  complete  the  following  ques¬ 
tionnaire. 


TEST  GROUP* 

RADIOLOGY  QUESTIONNAIRE 


You  have  been  involved  in  a  test  of  the  Remote  Medical  '.»im  -■  -  -  >n  (RMDS) 

to  determine  its  effectiveness  in  transmitting  x-ray  images,  Diffm.  transmitting 
modes  and  resolution  levels  have  been  used,  which  accounts  for  the  differences  in  image 
quality  between  the  various  sets  of  cases.  Much  of  the  time  required  between 
transmission  of  each  image  has  been  due  to  the  data  collection  procedure  for  these 
tests,  and  should  not  bias  your  opinion  of  the  system's  usefulness.  Keep  in  mind  that  the 
system  is  intended  to  provide  an  emergency  diagnostic  consultation  capability  between 
medical  personnel  at  remote  or  isolated  sites  and  larger  medical  facilities. 

We  would  like  you  to  take  a  few  more  minutes  to  answer  the  following  questions: 

1.  Do  you  feel  that  satisfactory  radiology  consultations  for  emergency  cases 
can  be  made  via  the  RMDS?  (Please  discuss.) 


2.  Do  you  feel  that  there  are  particular  types  of  pathologies  that  may  or 
may  not  be  readily  diagnosed  using  the  RMDS?  (Please  indicate  such 
types.) 


3.  Was  the  "zoom"  capability  (enlargement  of  a  portion  of  the  image)  useful 
to  you  in  making  a  diagnosis? 


•These  questions  were  presented  to  members  of  the  Test  Group.  However,  the  subjects 
were  not  advised  as  to  which  group  they  belonged  to. 


If  you  used  the  "zoom"  feature,  did  you  save  the  full-sized  image  using  the 
"video  storage"?  If  so,  was  that  useful  and  how? 


Would  one  image  be  sufficient  for  this  type  of  system,  or  are  two 
simultaneous  images  required?  (Please  discuss.) 


Did  you  use  the  "reverse  polarity"  feature  (positive  or  negative  image)? 
Would  such  a  feature  be  of  any  use  to  you  as  a  radiologist? 


Do  you  see  a  need  for  archiving  some  images  on  disc  memory  for  later 
consultation? 


Please  make  any  additional  comments  you  wish. 


CONTROL  GROUP* 
RADIOLOGY  QUESTIONNAIRE 


You  have  been  involved  in  a  test  of  the  Remote  Medical  Diagnosis  System  (RMDS) 
to  determine  its  effectiveness  in  transmitting  x-ray  images.  Different  transmitting 
modes  and  resolution  levels  have  been  used,  which  accounts  for  the  differences  in  image 
quality  between  the  various  sets  of  cases.  Much  of  the  time  required  between 
transmission  of  each  image  has  been  due  to  the  data  collection  procedure  for  these 
tests,  and  should  not  bias  your  opinion  of  the  system's  usefulness.  Keep  in  mind  that  the 
system  is  intended  to  provide  an  emergency  diagnostic  consultation  capability  between 
medical  personnel  at  remote  or  isolated  sites  and  larger  medical  facilities. 

We  would  like  you  to  take  a  few  more  minutes  to  answer  the  following  questions: 

1.  Do  you  feel  that  satisfactory  radiology  consultations  for  emergency  cases 
can  be  made  via  the  RMDS?  (Please  discuss.) 


2.  Do  you  feel  that  there  are  particular  types  of  pathologies  that  may  or 
may  not  be  readily  diagnosed  using  the  RMDS?  (Please  indicate  such 
types.) 


3.  Was  the  "zoom"  capability  (enlargement  of  a  portion  of  the  image)  useful 
to  you  in  making  a  diagnosis? 


♦These  questions  were  presented  to  members  of  the  Control  Group.  However,  the 
subjects  were  not  advised  as  to  which  group  they  belonged  to. 
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Would  one  image  be  sufficient  for  this  type  of  system,  or  are  two 
simultaneous  images  required?  (Please  discuss.) 


Do  you  see  a  need  for  archiving  some  images  on  disc  memory  for  later 
consultation? 


Please  make  any  additional  comments  you  wish. 


